Moving imager camera for track and range capture

ABSTRACT

A precision motion platform carrying an imaging device under a large-field-coverage lens enables capture of high resolution imagery over the full field in an instantaneous telephoto mode and wide-angle coverage through temporal integration. The device permits automated tracking and scanning without movement of a camera body or lens. Coupled use of two or more devices enables automated range computation without the need for subsequent epipolar rectification. The imager motion enables sample integration for resolution enhancement. The control methods for imager positioning enable decreasing the blur caused by both the motion of the moving imager or the motion of an object&#39;s image that the imager is intended to capture.

CROSS-REFERENCE TO RELATED APPLICATION

This is a Divisional application of prior application Ser. No.08/989,202 filed on Dec. 11, 1997, now U.S. Pat. No. 6,693,666, whichclaims the benefit of U.S. Provisional Patent Application No.60/032,761, entitled “MOVING IMAGER CAMERA FOR TRACKING, SCANNING, RANGEAND SUPER-RESOLUTION, ” filed Dec. 11, 1996, which is incorporatedherein by reference in its entirety. This application is related toWoodfill et al.'s copending U.S. patent application Ser. No. 08/839,767,filed Apr. 28, 1997, entitled “Data Processing System and Method,” whichis incorporated herein by reference in its entirety.

TECHNICAL FIELD

This invention relates generally to computer input devices, and moreparticularly to digital image capture devices used to provide rangingand tracking information for a computer system.

BACKGROUND ART

The range of an object, i.e. the distance to the object from anobservation site, can be determined by the analysis of two or morespatially separated images (often referred to as “binocular images” whenthere are two images) that are taken from the observation site. In rangecomputation from simultaneously acquired binocular digital images, thearea of processing is limited to the visible region of overlap betweenthe two images. To maintain a reasonable region of overlap usuallynecessitates redirecting the optical axes of the cameras (i.e. changingtheir vergence) which introduces other problems including a usualnecessity to resample the imagery. As is well known to those skilled inthe art, “vergence” means the angle between the optical axes of thelenses.

The processing of binocular or multi-view imagery for range computationis easiest when the optical axes are parallel and the imaging surfacesare coplanar—in what is termed parallel epipolar geometry. Becauseverging the optical axes to optimize the region of image overlapeliminates image-surface co-planarity, the complexities of calculatingrange increases significantly with non-parallel viewing. This is furthercompounded when viewing objects at a variety of azimuths and distanceswhere adjustments in the view direction as well as verging would benecessary to retain sufficient image overlap.

For computational purposes, the frame of reference for scene descriptionis usually tied to image location, so changing the image locationthrough vergence adjustments necessitates reconfiguring the frame ofreference. Again, adjusting a system's frame of reference increases thecomputational and conceptual complexity of its analysis.

A similar situation arises for typical monocular (i.e. single image)computer analysis of tracking and scanning in some space before thecamera. With subjects able to operate over a broad region before thecamera, continued observation generally involves use of eitherwide-angle optics or a panning/tilting mechanism to properly direct thecamera's view direction. These control mechanisms are relativelycomplex, must move fairly large pieces of equipment (cameras and theirlenses), and alter the underlying geometric frame of reference for sceneanalysis by rotating the frame of reference with the cameras. Inaddition, the use of wide angle optics works against high resolutionanalysis, as only larger scene detail is visible.

One approach to solve these acquisition problems in image-based rangeand tracking computation would be to employ greatly oversized imagers(e.g. imagers having about 3K by 3K or 9×10⁶ elements), and selectcorresponding standard-sized windows within these for processing.However, such an approach would be prohibitively expensive. For example,a 1K by 1K imager sells for well over a thousand dollars. Higherresolution imagers are available at considerably greater price.

A prior art solution to the apparent dichotomy between simple processing(with parallel epipolar geometry) and broad depth and tracking coverageexists in adaptation of perspective-correcting lens systems as used in“view-camera” or “technical-camera” designs. In such designs, anoversized lens is used to image the scene, and lateral repositioning ofthe lens or imaging platform can be used to redirect the camera withoutrotating the imaging surface. For single camera use this enablesmaintaining lines parallel in the world parallel on the image plane; inranging camera use it enables parallel epipolar geometry.

For example, in U.S. Pat. No. 5,063,441 and 5,142,357 of Lipton et al.,devices for use in 3D videography are disclosed. More particularly,Lipton et al. teach devices for capturing binocular images for use instereo videography (stereo movies), with reduced viewing eyestrain, byusing dual and triple camera system. Briefly stated, Lipton et alteaches an imager controller for epipolar stereo capture in videography,including stereo lenses mounted fixedly together in a single housing.Stereographics Inc., of San Raphael, Calif., produces a productembodying elements of the Lipton et al. patents.

In the matter of two-dimensional imager control, U.S. Pat. 5,049,988, ofSefton et al. teaches a system that provides the display of a videocapture window for surveillance applications. Phillips, in U.S. Pat. No.4,740,839, teaches a TV surveillance system operated by sub-sampling aconventional camera, with a result that resembles the Lipton et al.approach of image capture.

As will be appreciated, image capturing of the prior art uses planarsensors due to the high cost, lack of availability, and complexitiesinvolved with the use and manufacture of curved or “spherical” sensors.However, spherical sensors have a number of advantages with respect tofield of view, view direction, and use in stereo image capture thatdesigners of prior art digital imaging cameras have apparently failed toconsider.

DISCLOSURE OF THE INVENTION

The present invention includes a multi-image camera system for automatedstereo ranging applications that provides high-resolution broad fieldcoverage over a wide range of subject distances while enabling retentionof the epipolar constraints necessary for efficient stereo matching. Atthe same time, the multi-image camera of the present invention supportsmonocular image applications such as object tracking.

In one embodiment of the present invention, an imaging device ispreferably placed on a three-degree-of-freedom linear motion positioningmechanism whose position can be controlled by a computer. Broad fieldcoverage is preferably attained through use of a wide-anglelarge-coverage lens (e.g. a lens designed for a 35 mm camera). Highresolution is attained through placing the imaging device under thelarge-coverage lens, so the imaging device's immediate field of view isconsiderably narrower than that provided by the lens.

In contrast with traditional view-camera usage, the present inventionteaches moving the imaging surface instead of the lens. This allows theapparatus to retain co-planarity of the images and, so long as thedisplacements can be determined to sub-pixel accuracy, maintains astable frame of reference for the analysis, all while providing therequired view re-directions. Moving the lens alters the projectiverelationships among observations, whereas moving the imager does not.

Image focus may be attained through the traditional rotating-travelfocus adjustment although, for quantitative computational tasks, this isoften unsatisfactory as the center of projection may vary with lensrotation. For this situation, our preferred embodiment of the movingimager provides a back-focus capability, as will be described below.

Computer control of the platform enables positioning accuracy in themicron range. Stepper motors, for example, operating with full steppositioning, can position a platform along an axis to within a fewmicrons of the desired position. Differential stepper control enablesincreasing the accuracy of this placement (the number of locations wherethis precision is attainable) by one to two orders of magnitude. Finespecification of absolute imager location can also be attained throughuse of interferometers or sonar-based ranging systems, as will bedescribed below in more detail.

The planar imaging embodiment of the present invention enjoys a numberof advantages over related devices of the prior art. In the presentinvention, an oversized lens is used to provide a wide field of view ofthe scene. Movement of the imaging surface under the lens provideshigh-resolution view selection, which is the equivalent of viewredirection of pan and tilt motions without the need for a complexmechanism for accomplishing such motions. With accurate positioningknowledge, this form of imaging provides a stable frame of reference forscene computation. Accurate positioning information may be attainedusing the positioning and control systems described below.

Furthermore, the apparatus of the present invention provides aneconomical solution to the aforementioned problems of the prior art. Forexample, the present invention is operable with relatively economicalhigh-precision imagers that use displacement for direction selection intracking, scanning, and range computation. More particularly, certainembodiments of the present invention place an imaging device on athree-degree-of-freedom linear motion platform whose position can becontrolled by a computer to efficiently and economically provide thedesired direction selection. The third degree of freedom providesback-plane focus control. Back-plane focusing has an advantage fortraditional lens designs where focus from element rotationaldisplacements causes image centering variations.

In a spherical camera embodiment of the present invention, image captureis effected through spherical focal-plane imaging using, for example, aspherical-faced fiber-optic faceplate. One lens suitable for thisembodiment, the Baker Ball lens, is described in a wartime reportreferenced below. In this spherical-imaging embodiment of the presentinvention, a mechanism is provided that shares many of the designconsiderations of the above-described linear moving-imager camera.Preferably, this embodiment uses a high resolution imaging device (1K by1K elements with 12 bits of resolution at each pixel), mounted behind afiber-optic faceplate that transfers the focal-plane light pattern tothe sensor elements of the imager. Notable in this embodiment of themoving imager is that the lens system has no particular optical axis oraxis along which any imaging surfaces are preferentially oriented.

In addition, the spherical camera embodiment of the present inventionhas a number of advantages over prior art planar imaging cameras. Forexample, the spherical camera embodiment we describe has 1) excellentoptical resolution attainable since the focal surface does not need tobe made planar (the lens is diffraction limited); 2) greater simplicitydue to the advantages of rotational over linear displacements; 3) agreatly increased undistorted field of view; 4) exponentially less lensillumination fall-off with eccentricity (cosine of radial displacementrather than the fourth power of the cosine); 5) greater effective panand tilt velocities; and 6) opportunity to study (via simulation)aspects of human psychophysical behavior in a mechanism of similargeometry.

Certain embodiments of the present invention, both of the linear andspherical lens, provide lenses in multiple housings to increase theflexibility, reliability, and range finding quality of the system. Amonocular version of the present invention requires only a single lens,and multi-lens versions of the present invention can use two or morelenses for multi-image ranging analysis.

These and other advantages of the present invention will become apparentupon reading the following detailed descriptions and studying thevarious figures of the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagrammatic view of a monocular imager in accordance withthe present invention;

FIG. 2 is a diagrammatic view of a multi-camera (binocular in this case)imager in accordance with the present invention;

FIG. 3 is an illustration of an imager under a wide angle lens of thepresent invention;

FIGS. 4 and 5 illustrate the use of a binocular imager in adjusting therange of available stereoscopic images in accordance with the presentinvention;

FIGS. 6 and 7 illustrate an alignment mechanism in accordance with oneembodiment of the present invention;

FIGS. 8 and 9 provide diagrammatic views of a multi-camera imagerwherein each moving imager utilizes a three platform positioningmechanism; and

FIG. 10 is an illustration of a spherical imager of the presentinvention.

BEST MODES FOR CARRYING OUT THE INVENTION

With reference to FIG. 1, a monocular imager apparatus or moving imager“camera” 10 in accordance with one embodiment of the present inventionwill now be described. The moving imager camera 10 includes a lens 12,and an imaging array, imaging device, or “imager” 14. As indicated bythe arrows 16, 17, 18, 19, 100, and 102, the imager is capable ofindependent motion along the three degrees of freedom defined by anX-Y-Z axis. This means that the imager 14 can move both in-plane andout-of-plane with respect to the lens 12 for view selection and focus,and may also be synchronized for rectangular displacements 20 in theimager plane P to provide super-resolution frame integration. Thedisplacements 20 are preferably rectangular, although they arerepresented by a circle in this figure. As will be appreciated by thoseskilled in the art, “rectangular” means any appropriate movement of theimager in the plane P to provide the necessary local sampling. Forexample, these rectangular type displacements are described in Tagami etal.'s U.S. Pat. No. 5,402,171, issued Mar. 28, 1995.

Preferably the motions of the moving imager camera 10 are controlled bya computer controller 21 through the use of suitable transducingmechanisms. While high precision encoders or the like may be workable,it is preferable that an extremely high precision measurement systemsuch as one utilizing an interferometer or sonar based mechanism beused. For imparting motion, an actuator such as a stepper motor issuitable. A variety of commercially available actuators are suitable forimparting motion along the X, Y and Z axes. For example, a steppermotor, along with appropriate gearing and linkages, available fromPhysik Instruments of Germany (with suppliers in the U.S.) has beenfound to be suitable for imparting these motions. Those skilled in theart will be familiar with the design and implementation of the variousactuators, high precision measurement systems, and the necessarycomputer control. A few illustrative embodiments of actuators andmeasurement systems suitable for controlling and determining theposition of the imager 14 will be described in more detail below.

The lens 12 can be a relatively inexpensive, wide angle lens,commercially available from a variety of sources. For example, a 35millimeter (“mm”) camera lens works well for the present invention. Onesuitable imager 14 is an imaging array having a resolution of at least512×512 pixels, although imaging arrays of lesser resolution will work.Such imaging arrays are available from a variety of sources, includingSharp Electronics, Inc. of Japan (with suppliers in the U.S.). Anothercontemplated imager 14 is a photosensitive integrated circuit having anarray of photosensitive elements. One advantage in this particularembodiment is that significant processing of the signals generated atthe photosensitive elements could be performed locally within thephotosensitive integrated circuit.

In FIG. 2, a multi-camera or “binocular” version of the presentinvention is illustrated. More particularly, an imager apparatus or“camera” 22 includes a pair of independent monocular apparatus 10 a and10 b (see FIG. 1 for corresponding elements) that are operated inconcert. However, it should be noted that any number n of monocularapparatus 10 can be used, in concert, for particular applications. Forexample, n=3, 4, 5, or 6 are useful for certain applications.

The apparatus 10 a includes a lens 12 a, and an imaging array or“imager” 14 a. As noted by the arrows 16 a, 17 a, 18 a, and 19 a, theimager 14 a is capable of independent in-plane motion with respect tothe lens 12 a for view selection. In certain embodiments, the imager 10a may also move in the out-of-plane direction for focus control. Inaddition, the imager 14 a is preferably synchronized for rectangulardisplacements (as described above with reference to FIG. 1) 20 a in theplane Pa of the imager to provide super-resolution frame integration.

The various components of the apparatus 10 a (e.g. the lens, motionactuators, imager, etc.) are as described previously with respect to theapparatus 10 of FIG. 1.

Similarly, the apparatus 10 b includes a lens 12 b, and an imaging arrayor “imager” 14 b. As noted by the arrows 16 b, 17 b, 18 b, and 19 b, theimager 14 b is capable of independent in-plane motion with respect tothe lens 12 b for view selection. In certain embodiments, the apparatus10 b may also move in the out-of-plane direction for focus control. Inaddition, the imager 14 b is preferably synchronized for rectangulardisplacements (as defined above) 20 b in the plane Pb of the imager toprovide super-resolution frame integration. The various components ofthe apparatus 10 b (e.g. the lens, motion actuators, imager, etc.) areas described previously with respect to the apparatus 10 of FIG. 1.

The apparatus 10 a and 10 b of camera 22 operate as follows. The lateralmotions 17 a and 19 a of apparatus 10 a and the lateral motions 17 b and19 b of apparatus 10 b are independent so that the two devices can bemoved for pan and vergence. However, vertical motions 16 a and 18 a ofapparatus 10 a and 16 b and 18 b of apparatus 10 b are coupled togethereither mechanically or through computer control for tilt control, aswill be appreciated by those skilled in the art. Likewise, therectangular displacements 20 a and 20 b are synchronized forsuper-resolution integration. These independent and synchronizeddisplacements can be easily accomplished by computer control 21 a and 21b (each similar to computer control 21 of apparatus 10 of FIG. 1 andmutually communicating by a communication link 23) or by a unitarycomputer control 24.

In FIG. 3, a greatly enlarged field of coverage 10 of a lens 12 isillustrated. Each rectangle 26 represents a potential location of theimager 10 in the lens' field of coverage as intended for capture by 35mm film. “Additional areas” 28 are not typically used in filmphotography, but are accessible to the present invention. In fact, theentire area 12 can be sampled by the method and apparatus of thisinvention. For example, a ⅓ inch 4:3 imager provides 5 times lateral and5 times vertical coverage, for an effective field of 3200 by 2400 (ormore) for a nominal 640 by 480 pixel imager 10, selected as the“standard” image size under NTSC standards.

As noted previously, in the present invention, an imaging device isplaced on a three-degree-of-freedom linear motion mechanism whoseposition can be controlled by a computer. Relatively broad fieldcoverage is preferably attained through use of a wide-anglelarge-coverage (“wide,” e.g. 45 degree or larger field of view) lens.High resolution is attained by placing the imaging device under the widelens, so the sensor's immediate field of view is considerably smallerthan that provided by the lens.

For example, for a 6 mm-wide imaging device under a 50 mm lens, thelens' 45 degree field of view is reduced to approximately 45×(6/35)=7.7degrees at the sensor array, making it act as an approximately 400 mmlens at the sensor, and increasing pixel resolution by a factor of about6 in each direction. Super-resolution integration techniques can morethan double this to a factor of about 12 in each direction, as will beappreciated by those skilled in the art.

Computer control of the motion platform upon which the sensor arrayresides enables positioning accuracy in the micron range. Stepper motorsand other precision actuators for accomplishing this control are wellknown to those skilled in the art. Stepper motors operating with fullstep positioning can position a platform at discrete positions along anaxis to within a few microns. Differential stepper control enablesincreasing the accuracy of this placement by one to two orders ofmagnitude.

For fine determination of absolute imager location, a pair ofinterferometer or sonar-based ranging systems can be provided thatmeasure the positions of the sensors in the focal plane to sub-micronprecision during image capture. Since each imaging cell is about 10microns across in a current embodiment, this enables sufficientsub-pixel precision to keep the range estimates coherent across theoperational field of view. In the present example, interferometricmeasurement and differential stepper control may provide better than 3bits of sub-pixel localization over the range of motion

With reference to FIGS. 4 and 5, a method is described for utilizing amulti-camera in accordance with the present invention in order to adjustthe range of available stereoscopic imaging. FIG. 4 illustrates amulti-camera 22 including a pair of monocular apparatus 10 a and 10 b.The corresponding elements of apparatus 10 a and 10 b are describedabove with reference to FIGS. 1 and 2. In FIG. 4, the imager 14 a iscentered about a focal axis of the lens 12 a and the imager 14 b iscentered about a focal axis of the lens 12 b. Given this arrangement,the bounds (at least along the plane of the paper) of the view selectionof the imager 14 a are indicated by the lines 110 and 112, and thebounds (at least along the plane of the paper) of the view selection ofthe imager 14 b are indicated by the lines 114 and 116. The hatchedregion 118 indicates range of available stereoscopic imaging availablefor the arrangement of FIG. 4.

In FIG. 5, the imager 14 b has been repositioned along the X axis and,as a result, the range of available stereoscopic imaging available forthe arrangement of FIG. 5 includes not only the hatched region 118, butnow includes the hatched region 120. By making similar adjustments alongthe Y axis, the range of available stereoscopic imaging can be varied asdesired.

An important design consideration for the present invention is therealization that the alignment of multiple imagers for range computationis important (recall the discussion of parallel epipolar geometry in thebackground). However, it is contemplated that the multiple imagers maybe built as independent components and arrayed as desired for particularranging situations. Accordingly, the present invention teaches analignment mechanism operable to align the multiple imagers in order tosubstantially meet the desired parallel epipolar geometry. One preferredalignment mechanism that will enable rapid alignment includes a laserdiode system directed parallel to the imaging surface of and exiting afirst imager housing the laser diode. When the imagers are properlyaligned, the beam enters another imager, is reflected on to the nextimager, and so on, such that the beam returns in a single plane oftravel to the originating source. In this way, visual alignment ofmultiple imagers can be obtained to the sub-millimeter level with simpleintegrating signal processing and optimization at the detector elements,as will be appreciated by those skilled in the art.

With reference to FIGS. 6 and 7 (top view and side view, respectively),an alignment mechanism 150 for aligning two moving imagers in accordancewith one embodiment of the present invention will now be described. Thealignment mechanism 150 is jointly housed within moving imagers 152 and154 and includes a laser diode 156, mirrors 158 and 160, and aphotosensor 162. The laser diode 156, the mirrors 158 and 160, and thephotosensor 162 are arranged such that, when the moving imagers 152 and154 are properly aligned, a beam of light 164 generated at the laserdiode 156 is reflected within the moving imager 152 and directed back tothe photosensor 162. Thus proper alignment can be achieved by adjustingthe moving imagers 152 and 154 until the light beam 164 is detected atthe photosensor 162. Mechanical position adjusters, such as thoseavailable from Newport Inc. of Irvine, Calif., may be used to effect theimager alignment.

Note that the alignment mechanism 150 locates the laser diode 156,mirrors 158 and 160, and the photosensor 162 within the imagers, asopposed to locating them externally. Doing so enables better alignmentthrough properly machining the light paths internal to the imagers. Aswill further be appreciated, the strategy of FIGS. 6 and 7 can beadapted to align a multiplicity of imagers using a single laser diodeand additional mirrors.

Another design consideration for the present invention is therealization that the precise measurement of location of the sensor ismore important than positioning accuracy. For most view selections,there is no need to achieve a particular exact direction of view: anapproximate direction will be sufficient. In other words, the sensordoes not need to be at a specific position under the lens and yet,wherever the sensor is placed, knowledge of its position is preferred toa quite high precision (e.g. a tenth of a micron). As an illustrativeexample, the sensor array could be located at an arbitrary position withrespect to the imaging plane, as long as the scene detail of interest isimaged and the system measures the imager's actual location with a highdegree of precision. The precision of this placement determines theaccuracy of the depth computed.

Yet another design consideration for the present invention is therealization that super-resolution computation is facilitated by havinghigh positioning accuracy over a distance of about ½ pixel (see Tagamiet al.'s U.S. Pat. No. 5,402,171). The alternative is quite acceptable(i.e. knowing to high precision where the other images contributing tothe integration are located). But, knowing the displacements in advanceand then using the known displacements during processing (for example,exactly one half pixel displacement in each direction) is a preferablesolution.

There are a number of enhancements to the basic methods and apparatusdescribed above with reference to FIGS. 1-7. It should be noted that thepresent invention includes an instantaneous wide-angle viewingcapability. Due to this wide-angle viewing capability, the requirementof moving the imager over the whole field of view while integrating toobtain a full view is eliminated. Instead, preferably through use of asmall positive-curvature converter lens between the lens and the imager,the full field of view is simultaneously projected onto thecentrally-located sensor.

Another enhancement to the invention is the inclusion of acomputer-controlled lens focusing mechanism, in addition to theback-plane focusing means provided by the platform out-of-plane motion,and this would preferably use stereo-computed range to determinefocusing distance. Another enhancement is a computer-controlled zoommechanism that will allow adaptive selection of effective field of viewover a wide range of values. The implementation of computer-controlledlens focusing and zoom mechanisms are well known to those skilled in theart.

Thus one particular embodiment of the present invention includes athree-dimensional-adjusting imaging camera that has three motionplatforms coupled for three-dimensional movement, a controller formoving the platforms separately or in concert, a measurement system fordetermining the location of the platforms in an external frame ofreference, an imaging device positioned for three-dimensionaldisplacements on the coupled platform, and a lens whose focal surfacecoincides with the imaging surface of the imaging device underthree-dimensional movement of the motion platform and which has a fieldof coverage considerably larger than the optically sensitive area of theimaging device.

A three-dimensional-adjusting imaging camera 200 in accordance with suchan embodiment will now be described with reference to FIGS. 8 and 9. Theimaging camera 200 includes a pair of positionable imaging apparatus 202a and 202 b that are operated in concert. The imaging apparatus 202 aincludes motors 204 a, 206 a, and 208 a, linear slides 210 a, 212 a, and214 a, an imager 216 a, and three platforms 218 a, 220 a, and 22 a.

The motor 208 a drives the platform 220 a along the linear slide 214 a,the motor 204 a drives the platform 218 a along the linear slide 212 a,and the motor 206 a drives the platform 221 a along the linear slide 210a. The imager 216 a is mounted upon a surface of the platform 221 a.Motion along the linear slide 214 a is perpendicular to motion along thelinear slide 212 a, but motion along both linear slides 212 a and 214 ais in-plane motion. Motion along the linear slide 210 a is out-of-planemotion. Hence, in-plane positioning is effectuated by actuating themotors 204 a and 208 a, while out-of-plane motion is effectuated byactuating the motor 206 a. Operation of the positionable imagingapparatus 202 b is similar and should be self-evident.

The camera can be used for the automated tracking of objects in thefield of view and the automated scanning of a field of view. Inaddition, the use of such a camera for integrating images from differentpositions can be used to develop higher resolution composite images ofthe field of view, and for the automated range computation to featuresand elements in the field of view. A lower-resolution version of thelatter is preferably accomplished with a converting lens positionedbetween a primary lens and imaging device surface such that a largeportion of the field of view of the lens is projected onto the opticallysensitive area of the centrally-located imager.

As noted, the present invention may further include a computercontrolled focusing mechanism aside from the back-plane focusing means.In addition, the camera of the present invention may further include acomputer-controlled focal-length adjusting mechanism.

A further advantage accruing from moving the imagers under computationalcontrol is that shuttering and acquisition can be synchronized to ensurethat images are free of motion blur—either from camera motion or fromobserved object motion.

The present invention has a great number of valuable applications inthat any application of computational video processing, includingobservation, tracking, and stereo ranging, would benefit from manyelements of the present camera design. These benefits include simplifiedstereo matching, the retention of a single frame of reference, highresolution and wide field of view with inexpensive off-the-shelf lenses,field of view selection without camera or lens movement, andsuper-resolution.

An alternative to the above-described imaging devices is an imagingdevice which utilizes a spherical lens. The human eye is the principalcapture-device model for theoretical aspects of computer visionresearch. Unfortunately, imager fabrication constraints have preventeddevelopment of imaging devices that can exploit the geometry of thisnatural “device.” Previously, researchers have typically only had accessto planar focal-surface sensors such as those described above withreference to FIGS. 1-9.

In FIG. 10, a camera 30 includes a spherical lens 31 centered on anarbitrary spherical coordinate system and having a lens focal sphere 32.The camera 30 further includes an elevation control axis 34 and anazimuth control axis 36. These axes may be implemented with curvedcontrol paths. A positioner 38 (shown here broken away for clarity) isattached to the axes 34 and 36 and is used to hold an imager 40 at theintersection of the axes. The actual position of positioner 38 is shownat 38′. A spherical-faced fiber-optic faceplate 42 is attached over theimager 40 to define the spherical image formation surface of the camera.An elevation controller 44 is coupled to the elevation control axis 34,and an azimuth controller 46 is coupled to the azimuth control axis 36.The elevation controller 44 and azimuth controller 46 are preferablycomputer controlled by a computer controller 48.

A curved focal surface, such as the retina and the spherical surface ofthe faceplate 42 described above, has a number of advantages forcomputational image processing, including: 1) excellent opticalresolution attainable when the focal surface needn't be made planar; 2)the simplicity of rotational over linear displacements; 3) a greatlyincreased undistorted field of view; 4) exponentially less lensilluminance fall-off with eccentricity; 5) greater effective pan andtilt velocities; and 6) opportunity to study (via simulation) aspects ofhuman psychophysical behavior.

With the present invention, the capture of spherical focal-plane imagerythrough, for example, the use of the spherical-faced fiberopticfaceplate 42 provides these and other benefits. The present inventionprovides a mechanism for exploiting this in a spherical-imaging camerathat shares many of the design considerations of the above-describedlinear moving-imager camera. The present design uses a high resolutionimaging device (1K by 1K elements with 12 bits of resolution at eachpixel), mounted behind a fiber-optic faceplate that transfers thefocal-plane light pattern to the sensor elements.

Rotational mechanisms allow azimuth and elevation controls on theimager-faceplate's position on a sphere, with mechanisms similar tothose used in our linear-motion device described above for measuringthis position to subpixel precision.

A suitable lens for this embodiment of the present invention is a balllens, which is described in J. Baker's article entitled “SphericallySymmetrical Lenses and Associated Equipment for Wide Angle AerialPhotography,” found in § 16.1 of the Nov. 30, 1949, REPORT OF UNITEDSTATES OFFICE OF SCIENTIFIC RESEARCH AND DEVELOPMENT, NATIONAL DEFENSERESEARCH COMMITTEE, which is incorporated herein by reference.Attachment of a fiber-optic faceplate to such a lens for astronomicstudy is described in Park et al.'s article entitled “Realtime TrackingSystem for the Wide-Field-of-View Telescope Project,” found in SPIE VOL.1111, ACQUISITION, TRACKING, AND POINTING III, 196-203 (1989) and Lewiset al.'s article entitled “WFOV Star Tracker Camera,” found in SPIE VOL1478, SENSORS AND SENSOR SYSTEMS FOR GUIDANCE AND NAVIGATION, 2-12(1991), both of which are incorporated herein by reference.

A lens similar in design character, although with a narrower field ofview, is manufactured by the Optical Corporation of America under thename “Wide Field-of-View Star Tracker.” The curvature of the faceplateis selected by its radius from the center of rotation. Mapping of theresulting image geometry for binocular and motion disparity computationis straightforward, as will be appreciated by those skilled in the art.A smaller, lower resolution, and mechanically simpler version of thisstereo eyeball camera may be attained by placement of a single imagingsurface with fiber-optic faceplate behind each lens of the two-lenscamera. As those skilled in the art will appreciate, rotation of thelenses and imagers in this eyeball system will not affect the frame ofreference, as this imager's lenses have no single optical axis, and theinterocular baseline remains fixed.

As well as being used for range computation, the spherical-imagingembodiment of the present invention can be advantageously used to studythe human eye, examining elements of its motion, stereo, and evenmonocular processing: tracking, saccades, micro saccades, foveation, andother such aspects.

Another aspect of the present invention teaches a method forsynchronization of the moving imager with image capture in order todecrease blur resulting from the imager's movement.

The concept of a moving image-capture device presents a problem in thatmoving while capturing induces blur. If capture of blurred images is tobe avoided, control of the moving imager's motions must be synchronizedwith the image capture process. If this synchronization can be attained,it presents the possibility of minimizing blur in two forms. First, ifall platform motion can be performed in the interval between imagecapture and the start of the next integration period, then noplatform-induced motion blur will be observed—the imagers will bestationary while integrating. Second, if a moving object is to beimaged, then movements of the platform tailored to the motion of theobject will enable higher resolution—minimal blur—imaging of it andanything else moving with its velocity. The preferred embodiment of themoving-imager camera synchronizes image integration and capture tominimize these capture artifacts.

As will be appreciated by those skilled in the art, video imagersgenerally operate at fixed scan frequencies. A usual mode of captureinvolves having the imager acquire light energy for a period of time(the integration time), and then to stop the integration process andtransmit out the acquired information. The time between the stop ofcapture (the end of the integration) and the start of the nextintegration period presents an opportunity for platform motion. Thistime may be from six to tens of milliseconds, depending on the length ofthe next integration period. In certain embodiments of this invention,these timing signals are available to the moving imager controller, andplatform motion is effected in the period between the termination of oneintegration period and the beginning of the next.

In another embodiment of this invention, the control signals to beginand end image integration are controlled by the moving imager computersuch that integration is begun only after the intended motion iscomplete. Note that in the latter embodiment a fixed or standard imageframe rate may not be attained but, for computational purposes asopposed to videography purposes, fixed or standard frame rates are notessential. Here, the quality and character of the acquired imagery arethe important factors and, where a clear image is desired, the controlmethod described will enable this capture.

The information used to control use and generation of thesesynchronization timing signals may come from measurement devices such asthe interferometer or sonar-based systems described earlier, or fromprecalibration experiments in which the velocity and timingcharacteristics of the platforms are measured and tabulated forsubsequent use.

While this invention has been described in terms of several preferredembodiments, there are alterations, permutations, and equivalents whichfall within the scope of this invention. As will be appreciated by theabove descriptions and the drawings, the imager control methods andapparatus of the present invention provide a stable frame of referencefor monocular and multiple image computer analysis for high-precisiontracking, scene scanning, and range computation over a wide field ofview and with detail of interest at a wide variety of distances.However, it should be noted that those skilled in the art will realizethat there are alterations, permutations, and equivalents as fall withinthe true spirit and scope of the present invention.

1. A moving imager camera comprising: a first positioning mechanismcapable of three-dimensional movement: a first imaging device having animaging surface, the first imaging device being mounted upon a surfaceof the first positioning mechanism such that the first imaging devicemoves in concert with motion of the surface of the first positioningmechanism; a first measurement system operable to determine a positionof the first imaging device within an external frame of referencedefined by three axes X, Y, and Z; and a first lens having a focalsurface and a field of coverage, the first lens arranged such that thefocal surface coincides with the imaging surface of the first imagingdevice, the field of coverage of the first lens being larger than theoptically sensitive area of the imaging surface, a second positioningmechanism capable of three-dimensional movement; a second imaging devicehaving an imaging surface, the second imaging device being mounted upona surface of the second positioning mechanism such that the secondimaging device moves in concert with motion of the surface of the secondpositioning mechanism; a second measurement system operable to determinea position of the second imaging device in a focal plane associated withthe second imaging device; a second lens having a focal surface and afield of coverage, the second lens arranged such that the focal surfaceof the second lens coincides with the imaging surface of the secondimaging device, the field of coverage of the second lens being largerthan the optically sensitive area of the imaging surface of the secondimaging device; and an alignment mechanism that includes a laser diodedisposed within the first positioning mechanism and aimed at a mirrordisposed within the second positioning mechanism, and a light sensordisposed within the first positioning mechanism, such that when thefirst and second positioning mechanisms are aligned into substantiallythe same plane, a beam of light generated by the laser diode willreflect off the mirror and return for measurement and optimization atthe light sensor.
 2. A moving imager camera as recited in claim 1further including a control system operable to control motion of thefirst and second positioning mechanisms.
 3. A moving imager camera asrecited in claim 2 wherein the control system includes a firstcontroller arranged to control the first positioning mechanism and asecond controller arranged to control the second positioning mechanism.